Pronunciation Clustering and Modeling of Variability for Appearance-Based Sign Language Recognition

In the domain of sign language recognition from video, most approaches try to segment and track the hands and head of the signer in a first step and subsequently extract a feature vector from these regions [1, 2]. Because of possible occlusions between the hands and the head of the signer, noise, or brisk movements, segmentation can be difficult. Many approaches therefore use special data acquisition tools like data gloves, colored gloves, or wearable cameras. Furthermore, the words and phrases of sign language are expressed differently by different signers. Sometimes there are two or three different pronunciations for one word. The pronunciations differ in the visual appearance. In this work, we introduce a database of video streams for American sign language word recognition. The utterances are extracted from a publicly available database and can therefore be used by other research groups. This database, which we call ‘BOSTON50’, consists of 483 utterances of 50 words. One important property of this database is the large variability of utterances for each word. This database is therefore more difficult to recognize automatically than databases in which all utterances are signed uniformly. So far, this problem has not been dealt with in the literature on sign language recognition. To overcome these shortcomings we suggest the following novel approaches: